In computing, a pipeline refers a set of data processing elements connected in series, where the output of one data processing element is the input of the next one. An ingestion pipeline is an example of such a pipeline. An ingestion pipeline can ingest and process input data in a process flow involving various data processing elements connected in a predetermined order. An example of input data can be, for instance, data collected by a data gatherer or crawler.
Such an ingestion pipeline generally processes data in an asynchronous manner. That is, data is processed by the ingestion pipeline in a predetermined order in one direction and the process flow does not return any processing output (e.g., processed data) back out of the ingestion pipeline in another direction. Rather, the processed data would be provided or moved to another computing facility downstream from the ingestion pipeline. To this end, an ingestion pipeline is useful for ingesting input data from data source(s) and preparing output data ready for further processing, for instance, indexing by an indexer (e.g., an indexing engine or a search engine). However, no upstream computing facilities could leverage the data processing functionality of the ingestion pipeline as the ingestion pipeline is not built to return, and thus is not capable of returning, processing results upstream from the ingestion pipeline. In view of the foregoing, there is room for technical innovations and improvements.